Guided exploration in gradient based policy search with Gaussian processes

Applying reinforcement learning(RL) algorithms in robotic control proves to be challenging even in simple settings with a small number of states and actions. Value function based RL algorithms require the discretization of the state and action space, a limitation that is not acceptable in robotic control. The necessity to be able to deal with continuous state-action spaces led to the use of different types of function approximators in the RL literature. However when stochasticity is present int the environment and the state-action spaces are continuous and high dimensional, estimated value functions cannot represent exactly the true value function corresponding to a policy, which leads to convergence problems when the action-selection policy is built upon the estimated state-action values [Sutton and Barto, 1998]. Gradient based policy search algorithms are more suitable for these types of control problems. In policy gradient (PG) methods a parameterized policy is improved upon each step of the learning algorithm, the direction of improvement is given by the gradient of a performance function. Convergence is guaranteed at least to a local minimum , and PG methods are computationally simple: the explicit representation of a value-function is not required. Moreover the incorporation of domain-specific knowledge is easily achieved through the parametric form of the policy and the underlying controller. The introduction of exploratory behavior however is difficult and often plays an important role in the performance of the algorithms. In this article we investigate the benefits of a fully probabilistic estimation of the action-value functionQ(·,·) through Gaussian process regression from the perspective of efficient exploration in policy gradient algorithms. We focus on the on-line learning of control policies in continuous space-action domains where the system dynamics is unknown and the environment presents a high degree of stochasticity. We use a state-action value function approximated with a Gaussian process (GP) and develop ways to alter search directions based on the accuracy and geometric structure of the approximated value function. Our methods allow the introduction of guided exploration based on current optimality beliefs while at the same time preserving the on-policy nature of PG learning.